CRITERIA FOR RATIONAL SMOOTHNESS OF SOME 
SYMMETRIC ORBIT CLOSURES 



AXEL HULTMAN 

Abstract. Let G be a connected reductive linear algebraic group over C with 
an involution 8. Denote by K the subgroup of fixed points. In certain cases, 
the .ff-orbits in the flag variety G/B are indexed by the twisted identities 
l(6) = {8(w~ 1 )w | w £ W} in the Weyl group W. Under this assumption, we 
establish a criterion for rational smoothness of orbit closures which generalises 
classical results of Carrell and Peterson for Schubert varieties. That is, whether 
an orbit closure is rationally smooth at a given point can be determined by 
examining the degrees in a "Bruhat graph" whose vertices form a subset of 
(.(#). Moreover, an orbit closure is rationally smooth everywhere if and only if 
its corresponding interval in the Bruhat order on i(9) is rank symmetric. 

In the special case K = Sp 2n (C), G = SL2n(C), we strengthen our criterion 
by showing that only the degree of a single vertex, the "bottom one", needs 
to be examined. This generalises a result of Deodhar for type A Schubert 
varieties. 



1. Introduction 

Let G be a connected reductive complex linear algebraic group equipped with an 
automorphism 9 of order 2. There is a (9-stable Borel subgroup B which contains 
a ^-stable maximal torus T [2"0I §7] with normaliser N. Let K = G e be the fixed 
point subgroup. We may always assume 9 to be the complexification of the Cartan 
involution of some real form Gr of G. 

The flag variety X = G/B decomposes into finitely many orbits under the action 
of the symmetric subgroup K by left translations. A natural "Bruhat- like" partial 
order on the set of orbits K\X is defined by inclusion of their closures. Let V denote 
this poset. Richardson and Springer [T5j[16j defined a poset map <p : V — » Br(W), 
where Br(VU) is the Bruhat order on the Weyl group W = N/T. The image of 
tp is contained in the set of twisted involutions 1(9) = {w G W | 9(w) = w -1 }. 
In general, ip is neither injective nor surjective. For certain choices of G and 9, 
however, (p produces a poset isomorphism V = Br(t(f?)), where i(0) — {6(w^ 1 )w | 
w £ W} C 2(9) is the set of twisted identities and Br(-) denotes induced subposet 
of Br(M^). In Section[3l we shall make explicit under what circumstances this fairly 
restrictive assumption holds. Now suppose that if is such an isomorphism and let 
O w , w £ i(9), denote the closure of the orbit O w = tp^ 1 (w). In this article we 
express the rationally singular locus of the symmetric variety O w in terms of the 
combinatorics of l(9). 

With each w S i(9), we shall associate a Bruhat graph BG(w) with vertex set 
Iw = {u E t(9) | u < w}. Our first main result, Theorem 15. 8\ states that O lB is 
rationally smooth at O u if and only if v is contained in p(w) edges for all it < v < w, 
where p(w) is the rank of w in Br(t(#)). In particular, O w is rationally smooth if 
and only if BG(io) is p(w)-regular. This latter statement also turns out to be 
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equivalent to the principal order ideal Br(I M ) being rank-symmetric; see Theorem 
[El below. 

The assertions just stated generalise celebrated criteria due to Carrell and Pe- 
terson [6] for rational smoothness of Schubert varieties. We recover their results in 
the special case where G — G' x G' and 9(x, y) = (y, x). 

The main brushstrokes of our proofs are completely similar to those of Carrell 
and Peterson. Below the surface, however, their results rely on delicate connections 
between Kazhdan-Lusztig polynomials and the combinatorics of (ordinary) Bruhat 
graphs. Our chief contribution is to extend these properties to a more general 
setting. Very roughly, here is what we do: 

First, properties of l(0) are established that combined with results of Brion [5] 
imply a bound on the degrees in BG(w) that generalises "Deodhar's inequality" for 
degrees in ordinary Bruhat graphs of Weyl groups. 

Second, an explicit procedure, in terms of the combinatorics of t(0), for com- 
puting the "i?-polynomials" of [HI [21] is extracted from the correspondence V <-» 
l(9). Using this procedure we establish several properties of these polynomials 
(and therefore of Kazhdan-Lusztig- Vogan polynomials) and relate them to de- 
grees in the graphs BG(to). This generalises well known properties of ordinary 
Kazhdan-Lusztig polynomials and i?-polynomials and how they are related to or- 
dinary Bruhat graphs. 

The most prominent example where our results say something which is not con- 
tained in [6] is G = SL 2n (C), K = Sp 2ll (C). For this setting, we prove the stronger 
statement (Corollary 16. 7j) that the degree of the bottom vertex alone suffices to 
decide rational smoothness. That is, O w is rationally smooth at O u if and only 
if the degree of u in BG(w) is p(w). This is analogous to a corresponding result 
for type A Schubert varieties which is due to Deodhar [7j. Again, that result is 
contained in ours as a special case. 

Remark 1.1. After a preliminary version of this article was circulated, McGovern 
[13] has applied our results in order to deduce a criterion for (rational) smoothness 
in the case G = SL2„(C), K = Sp 2 „(C) in terms of pattern avoidance among fixed 
point free involutions. Moreover, he proved that in this case the rationally singular 
loci in fact coincide with the singular loci. 

Closures of symmetric orbits are interesting objects in their own right, but an- 
other important reason to study their singularities is their impact on representation 
theory. We outline this connection while describing one of our main tools, Kazhdan- 
Lusztig- Vogan polynomials, in the next section. 

In Section [3[ we make precise the assumptions on 9 for which our results are 
valid. Thereafter, the Bruhat graphs BG(u>) are introduced in Section 0] Our 
Carrell-Peterson type criteria for rational smoothness are deduced in Section [5] 
Finally, in Section [6l we prove that the bottom vertex alone suffices to decide 
rational smoothness when G = SL 2 „(C), K = Sp 2 „(C). 

Acknowledgement. The author is grateful to W. M. McGovern for many helpful 
discussions. 

2. KLV POLYNOMIALS AND REPRESENTATION THEORY 

In the present paper, the principal method for detecting rational singularities 
of symmetric orbit closures is via Kazhdan-Lusztig- Vogan polynomials. Here, we 
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briefly review some of their properties and establish notation. For more informa- 
tion we refer the reader to [12] or [21]. Our terminology chiefly follows the latter 
reference. 

Let V denote the set of pairs (0,j), where O G K\X and 7 is a isT-equivariant 
local system on O. The choice of 7 is equivalent to the choice of a character of the 
component group of the stabiliser K x of a point x G O. In particular, 7 is unique 
if K x is connected. Since O is determined by 7, we may abuse notation and write 
7 for (0,7). With each pair 7, <5 G TJ, we associate polynomials i? 7i a,P 7i a G l>[q]- 
The i?-polynomials can be computed using a recursive procedure which we refrain 
from stating in full generality here; see [21] Lemma 6.8] for details. A special case 
sufficient for our purposes is formulated in Proposition 15. II below. 

Let M. denote the free Z[q, q^ 1 ] module with basis V. For fixed 5 G V, we have 
in M. the identity 

j- 1(fl Ev«h= E (-D iW) - ,( ' y) 9- ,( ' r) i , 7^(«- 1 )^,7(9)/' 

7<5 /5<7<<5 

which subject to the restrictions P in — 1 and deg(P 7i a) < (1(5) — £(7) — l)/2 
uniquely determines the Kazhdan-Lusztig-Vogan (KLV) polynomials P 7j( j 21, Corol- 
lary 6. 12] Here, /(•) indicates the dimension of the corresponding orbit, and the 
order on V is the Bruhat Q -order [21] Definition 5.8]. 

KLV polynomials serve as measures of the singularities of symmetric orbit clo- 
sures; cf. [21] Theorem 1.12]. In particular, their coefficients are nonnegative. An- 
other consequence is the following: 

Proposition 2.1. Let < denote the order relation in V, i.e. containment among 
orbit closures. Given orbits V,0 G K\X with V < O, let 6 = (0,Cq), where 
Co is the trivial local system. Then, O is rationally smooth at some (equivalently, 
every) point in V if and only if 
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1 ifL = e Q , 

ifL^C Q , 

for all 7 = (Q, L) G V with V <Q<0. 

The gadgets just described are fundamental ingredients in the representation 
theory of Gr- Fix an infinitesimal character for Gr. Then, T> is in bijective corre- 
spondence with two families of GR-representations with this infinitesimal character. 
Given 7 G T>, there is the standard (g, ifu)-module ^(7) induced from a discrete 
series representation, and there is the irreducible (g, iiR)-module ^(7). The tran- 
sition between the two families is governed by the KLV polynomials. Namely, one 
has 

©(7) = Et-^^'^'/rM©^ 
7' 

where 6(7) and 6(7) denote the characters of ^"(7) and ^(7), respectively [T2ll21j . 



^Note that there is a typo which has an impact on the cited result. We are grateful to D. A. 
Vogan for pointing out that the displayed formula in the statement of 1211 Lemma 6.8] should 

D(5) = u-'WJ2(- 1 Y ( ~' ) - m K-f,s(u)~ ( . 

7 
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3. Restricting the involution 

Consider the set V = {g G G \ 9(g~ 1 )g G AT}. The set of orbits K\V/T 
parametrises K\X. In this way, the map V —> W given by g i— » 9{g~ l )gT induces 
the map ip : V ^ W which was mentioned in the introduction. Observe that the 
image of <p is contained in 1(9). 

Throughout this paper we shall only allow certain choices of 9. More precisely, 
we from now on assume that 9 obeys the following condition: 

Hypothesis 3.1. The fixed point subgroup K is connected. Moreover, if : V — > W 
satisfies f(vo) G o{9), where vq G V is the maximum element, i.e. the dense orbit. 

Remark 3.2. If G is semisimple and simply connected, then K is necessarily 
connected. This result is due to Steinberg [20l Theorem 8.1]. In some sense, the 
general situation can be reduced to the study of semisimple simply connected G; 
see [15] . 

Several consequences are collected in the next proposition. We let $ denote the 
root system of G, T and write R C W for the corresponding set of reflections. 

Proposition 3.3. Hupothesis \3.1\ implies the following: 

(i) The map ip yields a poset isomorphism V —> Br(t(0)). 

(ii) There is a unique K-equivariant local system, namely Cq, on each orbit 
O G K\X. In particular, the sets T>, K\X and i(9) may be identified, and 
the Bruhat Q-order on T> coincides with V and Br(t(0)). 

(hi) Let a G $ and denote by G a C G the corresponding rank one semisimple 
group. Then, we are in one of the following two situations: 

(a) The root a is compact imaginary. That is, G a C K . 

(b) The root a is complex (meaning 9(a) ^ a) and 9(a) + a g" $. 

(iv) IfreR, then 9(r)r = r9(r). 

(v) The poset 3t(l(9)) is graded with rank function p being half the ordinary 
Coxeter length. Moreover, p(w) — l(O w ) — i(Oid)- 

Proof. Assertion (i) follows from Richardson and Springer's [HI Proposition 9.16]. 

For (ii), the local system on Kx, x G X, is unique if the isotropy subgroup K x is 
connected. Under Hypothesis 13.11 Springer's [TT] Proposition 4.8] implies that K x 
is connected if the torus fixed point group T e is connected. Since K is connected, 
this follows from [14[ Lemma 5.1]. 

In order to prove (iii), suppose 9(a) = a but G a 2 K. Then, the corresponding 
reflection r a G R is in the image of ip by [181 Lemma 2.5(i)]. This image is, however, 
l{6) which does not contain any reflections. 

If 6>(a) ^aand,3 = Q + 9(a) G then 9(f3) = (3 and Gp % K by [T71 Lemma 
2.6]. This once again leads to the above contradiction. 

Concerning (iv), assuming 9(r) ^ r p~7l Lemma 2.5] implies that the dihedral 
group generated by r and 9(r) is either of type A\ x A\ or of type A2- If the latter 
were true, we would have 9(a) + a G $, where a is the positive root corresponding 
to r. This contradicts part (iii), and the claim is established. 

Finally, the first part of (v) follows from (iv) in conjunction with [TO, Theorem 
4.6]. The second is then immediate from [15l Theorem 4.6]. □ 

The following example allows us to consider many of our results as generalisations 
of statements about Schubert varieties. 
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Example 3.4. If G' is a connected reductive complex linear algebraic group and 
G = G' x G' , the involution 6 which interchanges the two factors makes K the 
diagonal subgroup. In this case, l(9) = T(0), so Hypothesis \3.1\ is satisfied. The 
poset Br(i(#) coincides with Br(W'), where W is the Weyl group of G' . There is a 
one-to-one correspondence between K-orbits in X and Schubert cells in the Bruhat 
decomposition of the flag variety of G' which preserves a lot of structure including 
the property of having rationally smooth closure at a given orbit. 

In addition to the setting in Example 13.41 there are a few more cases that satisfy 
Hypothesis 13.11 They are denoted A II, D II and E IV in the classification of 
symmetric spaces Gr/Kr given e.g. in Helgason The corresponding Weyl 

groups are A2 n +i, D n and Eq, respectively, with 6 in each case restricting to the 
Weyl group as the unique nontrivial Dynkin diagram involution. Types D and E 
could in principle be handled separately. In the former case, l(9) has a very simple 
structure (cf. [TQl proof of Theorem 5.2]), whereas the latter admits a brute force 
computation. Thus, the main substance lies in the A2 n +i case where Br(t(#)) is an 
incarnation of the containments among closures of Sp 2n (C) orbits in the flag variety 
SL2„(C)/B; see Example 10.4] for a discussion of this case. Nevertheless, we 
have opted to keep our arguments type independent regarding all assertions that 
are valid in the full generality of Hypothesis 13. II There are two reasons. First, the 
natural habitat for Theorems 15.81 and 15.91 is the general setting; no simplicity would 
be gained by formulating the arguments in type A specific terminology. Second, we 
hope that the less specialised viewpoint shall prove suitable as point of departure 
for generalisations beyond Hypothesis 13.11 

4. "Bruhat graphs" 

Let * denote the 0-twisted right conjugation action of W on itself, i.e. u * w = 
8(w~ 1 )uw for u,w G W. Then l(6) is the orbit of the identity element id G W . 
Recall that I w = {u G l{9) \ u < w}. 

Definition 4.1. Given w G let BG(w) be the graph with vertex set I w and an 
edge {u, v} whenever u = v * t ^ v for some reflection t G R. 

Notice that BG(m) is an induced subgraph of BG(w) if u < w. See Figure [T] for 
an illustration. 

We shall refer to graphs of the form BG(u>) as Bruhat graphs, because in the 
setting of Example 13. 4i they coincide with (undirected versions of) the ordinary 
Bruhat graphs in W introduced by Dyer [8]. 

Our next goal is to show that (the first part of) Brion's [5] Theorem 2.5] implies 
lower bounds for the degrees in a Bruhat graph. This essentially amounts to a 
reformulation of the relevant parts of [S] using our terminology. 

Lemma 4.2. Let w G l(Q) and u, v G I w , u ^ v. Write u ~ 6(x~ 1 )x for x G W. 
The following are equivalent: 

(i) {u,v} is an edge in BG(w). 

(ii) There are exactly two distinct reflections t G R such that u*t = v. 



2 The "usual" construction of DII would yield G = S0 2 „(C), K = S(0 2 „-i(C) X Oi(C)) S 
02n— l(C) so that K is disconnected. However, passing to the fundamental cover, we have G = 
Spin 2n (C), K = Spin 2n _ 1 (C) in agreement with Hypothesis 13. II 
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Figure 1. A picture of the Bruhat graph BG(u>) where w = 
S5S3S4S5S1S2S3S1 6 l{9) C A§. Here, Sj denotes the simple re- 
flection (i, i + 1) in the usual manifestation of A$ as the symmetric 
group Sg. The involution 9 sends Se—% to Sj. A vertex u £ J m is 
labelled by the indices of a sequence of simple reflections whose 
product x satisfies u = 9(x~ 1 )x. The straight edges indicate the 
covering relation of Br(i(#)). 

(iii) There are exactly two distinct reflections t £ R such that 9{x~ 1 )9(t)tx = v. 
If t is one of these reflections, then 9(t) is the other. 

Proof. The implication (ii) (i) is obvious. 

We have (iii) =>■ (ii), since 9(x~ 1 )9(t)tx — v if and only if u*r — v for r = x~ 1 tx. 

In order to show (i) => (iii), assume v = 9(x~ r )9{r)rx = 9(x~ 1 )9(t)tx, for r, t G 
R. In particular, t9(t) = r9(r). Dyer's [El Lemma 3.1] shows that (r,9(r) 7 t,9(t)) 
is a dihedral reflection subgroup of W. Since W is simply laced (which e.g. follows 
from part (iv) of Proposition 13.31 and inspection of finite type Dynkin diagrams), 
this subgroup must be of type A\ x A\\ A 2 is not possible since t9(t) = 9{t)t. Hence, 
{r,9(r)} = {t,9(t)}, and these two reflections are the possible candidates for t. 

□ 

We are now in position to bound the degrees of a Bruhat graph. Combining 
the first part of Brion's [5j Theorem 2.5] with part (iii) of Proposition 13.31 shows 
that the rank of a vertex v = 9(x~ 1 )x in BG(u>) is at most half the number of 
complex reflections (i.e. reflections that correspond to complex roots) t € R such 
that 9(x~~ 1 )9(t)tx < w. By Lemma [4721 this is precisely the degree of v in BG(w). 
We thus have the following fact: 

Theorem 4.3. For w £ i{0)> the degree of each vertex in BG(w) is at least p(w). 



CRITERIA FOR RATIONAL SMOOTHNESS 



7 



Remark 4.4. In the setting of Example [34] Theorem [43] specialises to "Deodhar's 
inequality" in W; see [U §6] and the references cited there. 

Lemma 4.5. If {u,v} is an edge in BG(?«), then either u < v or v < u. Further- 
more, v has exactly p{v) neighbours u such that u < v. 

Proof. Suppose u — 9(x~ 1 )x ^ v = u * t for some x £ W, t € R. Define reflections 
r = u~ 1 9(t)u and r = xtx . Using part (iv) of Proposition 13.31 we compute 

urt = v = 9{x~ )9(t)tx = 9(x~ 1 )t9(t)x = 9(x~ 1 )T9(x)ur = 9(x~ 1 )rxr = utr. 

Thus, t and r commute. Hence, {u, ut, 9{t)u, v} = u{r, t) and, by Dyer's [H Theorem 
1.4] the subgraph of the (ordinary) Bruhat graph on W induced by these four 
vertices is isomorphic to the Bruhat graph of the dihedral group on four elements. 
In particular, all pairs {a, b} C {u,ut,0(t)u,v} except at most one satisfy a < b or 
b < a. Since the map y i— > 9{y~ 1 ) is a poset automorphism of the Bruhat order 
which sends 9(t)u to ut, {9(t)u,ut} is the only incomparable pair. This proves the 
first assertion. 

For the second assertion, the above argument implies 

{teR\vt<v} = {teR\v*t<v}. 

It is well known that the left hand side has £(v) — 2p(v) elements. Lemma 14.21 
concludes the proof. □ 



5. A CRITERION FOR RATIONAL SMOOTHNESS 

In general, the recursion for the i?-polynomials mentioned in Section[2]is techni- 
cally rather involved. Since we are assuming Hypothesis 13. 11 however, the situation 
is simpler. Proposition [3T3] allows us to identify the indexing set V with l{9). With 
Dr(w) denoting the descent set of v 6 i-e. the set of simple reflections s such 
that vs < v, or equivalently v * s < v, the recursion takes the following explicit 
form: 

Proposition 5.1. Suppose s £ Dr(u). Then, the R- polynomials satisfy 

{Ru*s,v*s(q) ifu*S<U, 
qR u *s,v*s(q) + (q- i)R u ,v*s(q) if u* s > u, 
-Ru,v* s {q) ifu*s = u. 

Proof. Consider the free Z[g,(j _1 ] module M. with basis l{9). The definition of the 
map T s : M — > M formulated in [211 Definition 6.4] boils down to 

{qw if w * s = w, 

w * s if w * s > w, 

qw * s + (q — l)w if w * s < w, 

for w £ l(0) (the relevant cases being (a), (bl) and (b2), respectively). Equating 
coefficients in the identity 

(-l) p(u) RuA<l)u=- E (-l) piu) Ru, w *s(qKT s + l-q)u 
uQi{6) tiGt(8) 



s 
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(see [211 proof of Lemma 6.8]) now yields 

Ru*s,w* s (q) - Ru,w* s (q)(q -1 + 1-9) if « * s < 
Ru,w{q) = { qR u *s,w*s(q) - (1 - q)R u ,w* s (q) if u * s > u, 

■ju* s (q) iiu*s = u, 

if s s S satisfies w * s < w. 



□ 



Together with the "initial values" R u ,u{q) = 1 and R u ,v(q) = if u ^ v, we 
may calculate any R u _ v using Proposition ^. II Rather than working with the actual 
-/^-polynomials, we shall however find it more convenient to use the following simple 
variation: 

Definition 5.2. Foru,v 6 l{9), let Q u , v (q) = (-q) piv) ^ p(u) R^viq^ 1 )- 

One readily verifies the following recursion: 
Proposition 5.3. For s 6 D^(v), we have 

{Qu*s,v*s(q) ifu*S<U, 
qQ u *s,v*s(q) + (q - i)Qu,v* s (q) ifu*s>u, 
qQu,v* s (q) ifu*s = u. 

In particular, the Q u ,v(q) are polynomials. In the setting of Example 13.41 
both the R u ,v{q) and the Q u ,v(q) coincide with the classical Kazhdan-Lusztig R- 
polynomials introduced in [llj . The three lemmata coming up next hint that the 
Qu,v(q) may provide the more useful generalisation. 



Lemma 5.4. For u,v G t(#), we have 



1 if u < v and {u, v} is an edge in BG(«), 
otherwise. 



Proof. Suppose s £ Dn(v). Differentiating the equation in Proposition 15.31 with 
respect to q, and using that Q u ,v{l) = R u .v(l) = 5 U . V (Kronecker's delta), it follows 
that 

It is clear that {u * s, v * s} is an edge in BG(w) if and only if the same is true about 
{it, v}. Employing induction on p(v), it thus suffices to show that u * s < v * s if 
v * s ^ u < v and {u, v} is an edge. Lemma 14.51 shows that u * s and v * s are 
comparable in this situation. The assertion u* s > v* s would contradict the lifting 
property |10[ Lemma 2.7], and we are done. □ 

Lemma 5.5. Denote by /i the Mobius function o/Br(t(#)). Then, p(u,v) — Q u ,v{0) 
for all u, v £ t(#). 

Proof. Let us induct on p(v). The assertion holds for p(v) = because <3id,id(<7) = 
Rid,id(q) = 1- We shall demonstrate that p(u,v) satisfies the recursion for Q u ,v(0) 
derived from Proposition 15.31 
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Borrowing terminology from |10j . call [u, v] full if every twisted involution in the 
interval [u, v] is in fact a twisted identity. Combining Philip Hall's theorem (see 
e.g. [IH1 Proposition 3.8.5]) with [TD], Theorem 4.12] shows that 



fx{u,v) 



(_1)p(«)-p(«) if [ u , «] is full, 
otherwise. 



Pick s G Dn{v). In case u * s = u, [u,v] is not full, and fi(u,v) = as desired. If 
u * s > u, it follows from [10l Lemma 4.10] that [u, v * s] is full if and only if [u, v] 
is full. Thus, fi(u, v) — — v * s), and we are done. Finally, suppose u * s < u. 
If [u * s,v * s] is full then [u,v] is also full, again by [TQl Lemma 4.10]. On the 
other hand, [101 Theorem 4.9] implies that fx(u *s,v) = — v), so if [u * s, v * s] 
(and therefore [u * s, v]) is not full, then [u, v] cannot be full either. Completing the 
proof, we conclude (J,(u, v) — fi(u * s, v * s). □ 

Lemma 5.6. For all v € i(9), 

u<v 

Proof. We prove the lemma using induction on p(v). Given s 6 -Dr(i>), partition 
/„ into three sets: 

A={u<v\u*s< u}, 
B = {u<v\u*s> u}, 
C={u<v\u*s = u}. 

By the lifting property [101 Lemma 2.7], the map u i— ► u * s is a bijection between 
A and B. The recursion in Proposition 15 . 31 therefore yields 

(<?) 

(?) + (<7 - l)Q«,«*a(Q')) 

+ ^2 qQu, v *s{q) 
uec 

= <iQu,v*s{q)+ E (1 + 3- l)Q«,i>* s («) + E qQ u ,v*s(q) 

u<u.s u<u.s u<t)«a 

= q ^2 Qu,v*s(q), 

u<v*s 

proving the claim. □ 
Lemma 5.7. We have P u .v(0) = 1 whenever u < v in l(9). 

Proof. The assertion is clear if u = v, and we employ induction on p(v) — p(u). 
Vogan's [2lJ Corollary 6.12] translates to 

q p{v) - p{u) PuA<r 1 )= E QuAq)P w Aq)- 

u<w<v 



10 



AXEL HULTMAN 



The left hand side is a polynomial with zero constant term. Hence, Lemma 15.51 
implies 

P u ,v{0) = - ^ n(u,w) = [i(u,u) = I, 

u<w<v 

as desired. 

□ 

We are finally in position to prove the main results. Since all necessary technical 
prerequisites have been established, the corresponding arguments from [6] can now 
be transferred to our setting more or less verbatim. 

Theorem 5.8. Suppose u,v £ u <w. The following conditions are equivalent: 

(i) The degree of v in BG(jd) is p(w) for all u <v <w. 

(ii) The KLV polynomials satisfy P v . w (q) = 1 for all u < v < w. That is, the 
orbit closure O w is rationally smooth at O u . 

Proof. Define 

The P-polynomials have nonnegative coefficients. By Lemma 15. 71 fuwil) too is 
a polynomial with nonnegative coefficients. Since it has vanishing constant term, 
f'u = if and only if f u ,w(q) = which, in turn, is equivalent to P u ,w(q) = 1- 
Now, 

/; ji) = (pH ~ P {u)){PuAi) - 1) - 2p;ji). 

Since Q u ,w(X) — &u,w, w e have 
-2P; W (1) = |^(g- 2 )|,=i 

= 2(p(u)-p(w))P u , w (l) + 2 <9«,,(1)^(1) + 2< W (1). 

u<v<w 

Hence, 

= p(v) - P (w) + J2 QuA^PvM- 

U < V < w 

To begin with, assume (ii) holds. Then, 

P (w)-p(v)= Q'vA 1 ) 

v<v' <w 

for all u < v < w. Condition (i) now follows from Lemma 15 .41 together with Lemma 

EH 

Finally, let us prove (i) =>• (ii) by induction on p(w) — p(u). Suppose u < v < w 
in Bt(l(6)). By Lemma [5.41 and the induction assumption, Q' u V (1)P V , W (1) is one 
if {it, v} is an edge in BG(u>), zero otherwise. Since deg(u) = p(w), u has exactly 
p(w) — p(u) neighbours v such that u < v. We conclude f' u w (l) = as desired. □ 

Theorem 5.9. Form £ i(0), the following are equivalent: 

(i) The interval [id, w] — Br(/ U1 ) has equally many elements of rank i as of 
rank p(w) — i. 

(ii) The graph BG(u>) is regular. 

(iii) P u , w {q) = 1 for all u < w. 
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Proof, (i) => (ii): Let n(i) denote the number of elements of rank i in [e,ui]. Now, 
using Lemma 14.51 and Theorem l4.3[ we count the edges in BG(w) in two ways and 
obtain 

p(w) p(w) 
i=0 i=0 

with equality if and only if BG(w) is p(w)-regular. However, if n(i) = n(p(w) — i) 
for all i, then equality does hold. 

(ii) => (iii): This follows from Theorem 15.81 

(iii) => (i) : We claim that 

F w (q) = p u,w(<l)<l piu) 

is a symmetric polynomial, i.e. F w (q) = q p ^ F w (q^ 1 ). If the P-polynomials are all 
1, this means 



£ q p{u) = £?' 



u<w u<w 

It therefore remains to verify the claim. Observe that 



u<w 

= Y Y Qu,v(q)Pv,w(q) 

U<W U < V < w 

V<W u < v 

= F w (q), 

where the last equality follows from Lemma IBTBl □ 

To illustrate these results, consider Figure [TJ The interval [id, w] has three 
elements of rank three but only two of rank p(w) — 3 = 1. By Theorem 15.91 O w is 
rationally singular. A more careful inspection of the graph shows that S5S1 and e 
both have degree five whereas all other vertices have degree p(w) = 4. By Theorem 
15. 8( the rationally singular locus of O w therefore is S5Sl U O e . Also, observe that 
the degree never decreases as we move down in the graph. This phenomenon is 
explained in the next section. 



6. Sufficiency of the bottom vertex 

In this final section, the criterion given in Theorem l5.8l is significantly improved 
in the special case G = SL 2n (C), K = Sp 2n (C). In that case, as we shall see, 
whether or not an orbit closure O w is rationally smooth at O u is determined by the 
degree of u alone ( Corollary 16 . 71 below) . The corresponding statement for Schubert 
varieties is known to be true in type A [7] but false in general (see [4] for some 
elaboration on this). Necessarily, therefore, our arguments must be type specific 
since they cannot possibly be extended to the situation in Example 13.41 for arbitrary 

a. 
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6.1. Notation and preliminaries. Let us spend a few lines fixing notation with 
respect to the combinatorics of symmetric groups. 

We work in the set F% n of fixed point free involutions on {1, ... ,2n}. Let * 
denote the conjugation action from the right by the symmetric group S^n on itself, 
i.e. a * 7r = 7r -1 (77r. Then, F<i n = wq k S^n, where wq is the reverse permutation 
i \— ► In + 1 — i. 

If t = (a, b) £ S2n is a transposition and u £ i^n, then u k t = u if and only if £ 
is a 2-cycle in the cycle decomposition of u. If u*£ ^ u, the decompositions into 
2-cycles of u and w -kt are as follows: 

it = (a, it(a))(£>, ■ • • , 

u-kt = (a, u(b))(b, u(a)) • • • , 

where the dots denote the remaining 2-cycles (that both involutions have in com- 
mon). In particular, there is exactly one transposition t' ^ t such that u-kt' = u-kt, 
namely t' = (u(a),u(b)). 

Let if! denote the dual of the subposet of the Bruhat order on Sun induced by 
F2n- The bottom element of this poset is u)q. Observe that if u ^ u k t, then 
u k t >~ u iff t is an inversion of u (meaning t = (a, b) with a < b and u(a) > u(b)). 
If s = (i, i + 1) is an adjacent transposition, then s is a descent if it is an inversion; 
otherwise s is an ascent. 

For it £ F2 n , 1 < i,j < 2n, define 

U(i t j) — | {ir. £ {1, ... , 2n} | x < i and u(i) > j}\. 

Thus, uuj-s is the number of dots weakly northwest of in the permutation 
diagram of u. 

Lemma 6.1 (Standard criterion; Theorem 2.1.5 in [5]). For u,w £ F2 n , we have 
udiw iffu( itj -) > w^fi for all 

For w £ i^n, define the Bruhat graph BG(w) as the graph whose vertex set is 
Iw = {u £ F-m | u ^ w} and {u, v} is an edge iff u ^ v = ukt for some transposition 
t. Thus, each edge has exactly two transpositions associated with it, and the graph 
is simple (no loops or multiple edges). If w is understood from the context and 
u < w, let out(u) denote the set of edges incident to u in BG(w). Also, define 
deg(u) = |out(u)|. 

Proposition 6.2. Suppose W — Am-i — Sin with 9 : W — > W given by the 
unique nontrivial involution of the Dynkin diagram. Then, x i— > wqX defines a 
bijection Fm — * t{9)- Moreover, the bijection is an isomorphism of Bruhat graphs, 
i.e. u ■< w 4=> wqu < wqw and Wq(w kt) = wqw * t. 

Proof. This is immediate from the well known facts that 0(x) = wqxwq and that 
x i y wqx is an antiautomorphism of Br(M^). □ 

6.2. An injective map. Suppose w > u ^ wq and let r = (i,j), i < j, be a 
transposition such that uk r -i, u. Let a = u(i) and b = u(j). Thus, a < b =/= i. 

For a transposition t — (x,y), we use the notation supp(i) = {x,y}. 

Definition 6.3. A transposition t is called compatible (with respect to u and r) if 
either supp(i) n {a, b,i,j} — or supp(i) n {i, j} ^ 0. 

Given an edge e £ out(u) there are precisely two transpositions t and t' ^ t such 
that e = {u,ukt} — {u,ukt'}. At least one of them is compatible. 
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Definition 6.4. For any edge e £ out(u), let t e be a compatible transposition such 
that e = {u, u k t e }- 

Definition 6.5. Given e £ out(it), define e(e) = {uk r,uk rr e }, where 



The point of all this is the following: 
Theorem 6.6. Definition \6.5\ defines an injective map e : out(u) — > out(u*r). 



By Theorem 16.61 the degree can never decrease as we go down along edges in a 
Bruhat graph. In particular, if a vertex has the minimum possible degree, then so 
does every vertex above it: 

Corollary 6.7. We have deg(w) = deg(w) for all u ■< v ■< w if and only if 
deg(u) = deg(w). 

Thus, to determine whether condition (i) of Theorem 15.81 is satisfied, it suffices 
to check the degree of u. 

Remark 6.8. The set S2n = {w £ F% n \ i < n =>• w(i) > n + 1} is in natural bijec- 
tive correspondence with S^n i n a way which identifies Br(S , 2„) with ^. Restricted 
to w £ Corollary 16 . 71 specialises to a result of Deodhar [7] for type A Schubert 
varieties. In that setting, our arguments are closely related to work of Billey and 
Warrington §6] 

Remark 6.9. Observe that for G = SL 2n (C), K = Sp 2 „(C), Theorem 1431 follows 
directly from Theorem l6.6l Thus, we have reproven Brion's [5j Theorem 2.5] in this 
case. 

6.3. Proof of Theorem mi 

Lemma 6.10. The set e(e) is well defined, i.e. independent of the choice oft e . 

Proof. This is clear if supp(i) n {a, b, i,j} = 0. If not, the only case when both 
transpositions associated with e are compatible is when e = {u, u* t} = {u, u * t'} 
for {t, t'} = {(i, b), (j, a)}. In this case, we have u-ktr — u-kt'r and u*rt — u*r = 
u*rt'. □ 

Lemma 6.11. For every e £ out(u), we have e(e) £ out(u*r). 

Proof. We must show that u* r ^ u* rr e ^< w. 

First, assume u* r = u* rr e . Then, u-kt e = u * rr e rt e . If r e = rt e r , this means 
u-kt e = u which contradicts the fact that e £ out(u). If, on the other hand, r e = t e , 
then we conclude that r and t e do not commute, hence that rt e rt e r = t e . But then, 
u -k t e r — u-k rt e rt e r = u*t e < w which contradicts r e = t e . Thus, u-k r =/= u-k rr e . 

It remains to prove u k rr e ^ w, i.e. that either u k t e r ^ w or u k rt e ^ w (or 
both). There are a few cases: 

Case 1. If supp(£ e ) n {i, j, a, b} — 0, then ukt e (i) < ukt e (j). Thus, w >r ukt e >- 
u k t e r. 




rt e r if uk t e r ^ w, 
t e otherwise. 



Proof. This follows from Lemmata 16.101 16.111 and 16.121 below. 



□ 
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Case 2. 


If t e = 


(i, j), then u -k t e r — u < w. 


Case 3. 


Ht e = 


(i, b), then ukt e {i) = j so that ukt e r = ukt e < w. 


Case 4. 


lite = 


(j, a), we again have u k t e (i) = j. 


Case 5. 


If t e = 


(i, k) with k {j, a, b}, then rt e r = t e rt e = (j, k) and u*rt e = 


u * t e rt e r. 


Let c = 


= u(k). 



We have u*t e (i) — c, u*t e (j) = b, u*t e (k) = a, ukr{i) = b and ukr(k) = c. If 
k < j, it follows that w >u*t e > ukt e rt e r = u*rt e . Otherwise, k > i and either 
W h U*r y u*rt e (if b < c) or u k t e r -< u k t e (if b > c). 



Case 6. If t e = (j, k) with fc ^ {i, a, 6}, then rt e r — t e rt e — (i, k) and ukrt e = 
u k t e rt e r. Again, let c = u(k). 

Now, ukt e (i) — a, u-kte(j) — c and m * t e (k) = 6 so that ukt e r >- ukt e implies 
c < a < 6. It follows that either u * rt e — u* t e rt e r -< u k t e < w (if i < k) or 
m * r >- u * rt e (if fc < j). □ 

Lemma 6.12. If e ^ e' for e,e' 6 out(u), i/ien e(e) 7^ e(e'). 

Proof. Suppose e, e' G out(u) and e 7^ e'. There are three cases: 

Case 1. If T e = rt e r and r e > = rt e >r, then e(e) = e(e') ^ u-k t e r = ukt e >r <^> 
u * i e — u-kte' which contradicts e^e'. 

Case 2. Suppose r e = t e and r e / = t e >. Assume e(e) = e(e'), i.e. u*rt e — ukrt e >- 
If both t e and i e ' commute with r we argue as in the previous case. If not, since 
both t e and t e > are compatible, we either have {t e ,t e '} — {(i,b), (j,a)} leading to 
the contradiction u*t e — u-kt e >, or we have {t e ,t e >} = {{i,a), (j,b)} which implies 
the contradiction u = u*t e . 

Case 3. Finally, assume r e = rt e r and T e > = t e >. Then, the assumption e(e) = 
e(e') amounts to it*t e = u*rt e ir. Suppose rt e > 7^ t e >r and rt e 7^ t e r; otherwise 
we would be in Case 1 or 2, respectively. This implies that either rt e >r = t e or 
{t e ,rt e >r} = {(i,b),(j,a)}. The latter case, though, leads to u k r = ukrrt e >r — 
u*te'r implying the contradiction u = u * t e > ■ Thus, rt e >r = t e , and therefore 
{t ei te>} — {(i, k), (j, k)} for some k {i,j, a, b}. Let us suppose t e — {i,k); the 
other case is completely similar. A small computation shows that 

u = (i,a)(j,b)(k,c) 

u-kr = (i,b)(j,a)(k,c) • • • , 

u-kte = (i,c)(j,b)(k,a) • • • , 

u-krt e = (i,c)(j,a)(k,b) 

u-kter = (i,b)(j,c)(k,a) 

u k rt e r = (i, a)(j, c)(k, b) ■ ■ ■ , 

where we have written down the 2-cycle decompositions of the various elements 
(the dots indicate the remaining cycles; they are equal for all six elements). In 
particular, the elements are all distinct, so \u* (r,t e )\ = 6. 
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Now observe that precisely five of the elements in u* (r,t e ) belong to BG(w); 
the one which does not is u-krt e = u-kt e >r, because r e > ^ rt e >r. A contradiction is 
now provided by Lemma 16. 131 below. □ 

Lemma 6.13. Suppose \u-k(tx, £2} n Iw I > 5 for two elements u,w £ F 2n and some 
transpositions t\,t 2 . Then, \u * {t\, t 2 ) H I w \ = 6, i.e. U-k (t\,ti) QI W . 

Proof. The set of transpositions in the dihedral subgroup {t\,t 2 ) is {t\, t2, tit 2 ti} = 
{(xi, x 2 ), (x 2 , X3), (xi, X3)} for some 1 < x\ < x 2 < X3 < In. There are elements 
1 < o,\ < a 2 < a 3 < 2n, with Xi 7^ aj for all i and j, such that u * {t\,t 2 ) consists 
of the six involutions with cycle decomposition of the form 

(xi,a il )(x2,a i2 )(x 3 ,a i3 ) 

dots denoting the 2-cycles in u with support disjoint from {x\, x 2 , X3, 01, a 2l a^}. 
In order to simplify notation, let 

[«i«2«3] = (a;i,o il )(a:2,Oj 2 )(a;3,Oi s ) 

Since [123] is the maximum element in u-k (ii, t 2 ), it suffices to show that w >z [123] 
whenever w >z [213] and w >z [132]. To this end, consider the standard criterion. 
For 1 < a, (3 < 2n, let 

[hhh]~l atl3 ) = \{j 6 {1,2,3} I Xj < a and a tj > /?}, 

[hhhjfafl = \{j S {1,2,3} I a tj < a and x. ( > (3}. 

Then, the number of dots weakly northwest of (a, (3) in the diagram of [11^2*3] is 

[«i«2«3](q,/3) = [hizh^p) + 1*1*2*3]^^) 

where D counts dots with coordinates outside {x\, x 2 , X3, a±, a 2 , 03}; this number 
is independent of i\, i 2 , i%. 

By the symmetry between x and a, it is sufficient to show 

[123]+^ = mm([213]+^,[132]+ i/3) ) 

for all a, (3. This statement, however, follows immediately from the observation 
that for all m, the first m letters in the string "123" are the same as the first m 
letters in one of the strings "213" and "132". □ 
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